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BLUF 


RTRG: 

...brings near real-time intelligence to the 
warfighter 

..."grew up" supporting operations in Iraq 

...RTRG is now a global architecture 

...leveraging the emerging cloud architecture to 
answer questions we have not been able to do 
before 
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RTRG in Afghanistan 



0 



{S//REL} Area 82 - Bagram Air Base 
Home of RTRG AF1 & gmTote 


* 20 days data retention 
from AFPAK 

* 200-600 daily users 

* New upgrades include 
two systems in Kabul 

(TS//REL) 


Mission Areas: Tracking high-value targets (HVT), 
Counter-Insurgency (COIN), Counter-IED (CIED) 


Organizations Using RTRG: 

• CSG* Afghanistan 

• U.S. Marine Corps (USMC) 1 st and 2 nd Radio 
Battalions 

• U.S. Army SIGINT analysts at BCT* level 

• U.S. Air Force National Tactical Integration 

• Jalalabad Fusion Cell (USMC) 

• S2TOPI 

• NSA-G SWAN Counternarcotics Team 

• Ail special operations task forces 

(TS//SI//REL) 


"RTRG is the most significant SIGINT support to "USSOCOM has enduring and critical needs for 
the war fighter in the last decade" the tools and data that RT-RG provides" 

- General David Petraeus - Admiral William McRaven 


*CST- Cryptologic Support Team *BCT - Brigade Combat Team *CSG - Cryptologic Support Group 
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• In 2011, RTRG in Afghanistan 

- Played a key role in 90% of all 
SIGINT developed operations 

-Leading to 2270 
capture/kill operations 

-6534 enemies killed in action 


-1117 detaine 
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RTRG in the Gulf and Horn of Africa 





Monitoring Iranian Navy (IRIN) 
in Straits of Hormuz 


Photo of IRIN vessel from 
US Navy aircraft, located by RTRG 


Supporting CENTCOM Maritime (NAVCENT) 
Navy Information Operations Command- Bahrain (NIOC-B) 


RTRG Afloat on subsurface platform 
USS Georgia (SSGN-729) 


Missions supported: (ts//si//rel) 


Missions supported: [ts//si//relj 

• Iran, Yemen, Persian Gulf 


• Horn of Africa: In first week of mission. 

• Recent successes include monitoring of 


system received 31 million GSM events. 

Iranian naval assets 


leading to 10 high-value target voice ID, 



and 90 tactical tip-offs 
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RTRG Global Operations 


AF-1,5 Bagram & Kabul 

COINjCIED 


USS Georgia & 
USS Florida 


US-3 JFCOM 


BH-1 NIOC** Bahrain 


US-8 NSA-Texas 
Counternarcotics, Maritime, 
Mexico & SOUTHCOM Support 


Joint Forces Command AFRICOM, EUCOM, 

GE-3 Germany ** PACOM 2 USFK** Pangyo 

AFPAK PACOM & North Korea 

Continuity-of-Ope rations 


* In draw-down ** ECC - NSA European Technical Center USFK — U.5. Forces, Korea NIOC - Navy Information Operations Command 

TOP SECRET//SI//REL TO USA, FVEY 


US-2 - Denver US-1 Fort Meade 

Global Maritime & ELINT Mission Assurance 


IQ2 Iraq* 
COIN, Cl ED 
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• RTRG Mission Overview 

• RTRG System: Today and Tomorrow 

• Target-Centric and Network-Centric 
Cloud Analytics 

• Future Work 
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Current RTRG Architecture 


RTRG System 



Goldminer GeoT Agent Logic Sharkfinn 

— r 



G 


ijM 





SKS 


SKS 


Panopticon 

v 


Metadata Geospatial Alerting Selector Report & Doc Target 
Search Tools Tools Enrichment Manager Management 


Supporting Tactical Users 


Services Layer: Web services, authentication, auditing 



Publish and 
Subscribe 
Messaging 


Oracle relational database & 
dimensional data model 


Forward Data Centers 


Ingest and Enrichment Pipeline: 

Flexible, high-speed architecture for parser and data processors 









Data Feeds : drt 

CIDNE 

JUGGERNAUT 

LOPERS 

TAUS/MA TTERHORN 

AIRHANDLER LYCANTHROPE 

KL VOICESAIL 

.. and a 

RETURNSPRING 

growing list of others* 


(DNR, DIM I collect, tipping and reporting) 

A successful architecture for several years. Demand for more data feeds, 
longer retention, and data-intensive analytics has driven RTRG to seek new solutions 



* Based on Afghanistan RTRG data flow 
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RTRG Data Challenges 



Current Challenges 

• Data Storage St Retention 

- "Patterns of Life" analysis needs require 6+ months of data 
from world-wide collection 

— A typical system has capacity for only 4-6 weeks of regional data (~90% user 
queries are within seven days of "now") 

• Data Use St Computation 

— Analytic processes should make maximum use of all available data to find small 
signals 

- Relational databases are unsuited to sophisticated analytics such as correlation 
and matching 

• Data & Technology Heterogeneity 

- New types of data must be added to the system continually 

— With traditional databases, schema modifications are difficult 

— Exotic data management solutions are difficult to adopt due to limited expertise 
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Cloud Architectures for Analytics 




Emerging NSA Cloud Reference Architecture is well-suited for 
developing analytics on intelligence data 

Distributed file systems and databases are built 
Scalable: on clusters of commodity hardware, leveraging 
open source projects and industrial solutions 




The MapReduce programming model simplifies 
Computable : writing efficient parallel computations that 

operate over large volumes of data 



Scatabfe BigTabte Implementation with Security 


Flexible: 


Cloud technologies enable flexible schema and 
leverage large open-source efforts 


v 


j 
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Scalability & Computation 



Data Challenges in AFPAK 


Analytic Challenges from Iraq & AFPAK 


Current RTRG (AF1) 

Current database is 27 terabytes (TB) 
Retention is ~30 days 

Future Cloud enabled system 

Even a modest cloud system (3 rack) for 
storage will be at least 125 TB of storage 
5x increase in available space 
Actual retention improvement depends 
on how the space resources are allocated 


Cloud supports more data feeds & 
more days of historical data 


• Many analytics used by RTRG are 
based on R6 SORTINGLEAD 
event summaries 

• Event summaries were originally 
created on relational databases 

• Collection increased dramatically 
and a mapreduce implementation was 
needed 

• For new analytics with present day 
collection volumes, a practical parallel 
execution model is crucial 

Cloud supports large-scale analytics 


ii 
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NSA Cloud Computing Enterprise 



* 5-12 racks commodity hardware 

- 150+ data nodes 
16 GB RAM each 

* Apache Hadoop 

- 100s of terabytes of storage 

- Stores 10s to 100s of billions of events 

* NSA Cloudbase 

100s of nodes serving BigTable 

implementation 

Stores 100s of billions of entries 



gmUGHT 
gmBALTiC 
gm PLACE 
gmPARK 
gmHALO 
gmCARBON 


Europe 

■f V 

o , 

. 

gmROYALE 

gmREGAL gmGEiST 




Afghanistan 


gmTOTE 

(Bagram) 


Korea & Japan 
/gmPEN 


_ S. Korea 

> , 

l - 

J gmCALM 
Camp Humphreys 
S. Korea 



gmPAC 

Misawa, Japan 


Australia 



gmMATE 

(S//REL) (Alice Springs) 


GHOSTMACHINE is a data-intensive 
cloud system with many 
fielded instances worldwide 


Hadoop clusters have been 
demonstrated with 10s of petabytes 
and over 10,000 cores 


Map does not include all GHOSTMACHINE/SiteStore systems 
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Today's "Hybrid" Architecture 



gT 


C2 


** • ** 


m 



SKS 



f<NI VfU IHS 


Applications 



Services Layer 





* 




Publish and 
Subscribe 
Messaging 



Relational 

Database 






Ingest and Enrichment 


Large scale data 
preprocessing and 
analysis 





CJoildiiiES 

Scalabie Implementation with Security 

Massive storage and 
fast record lookup 



J L. 


GHOSTMACHINE 


RTRG and GHOSTMACHINE systems are paired with one another: 
MapReduce analytic results are fed back to RTRG relational database 
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User 

interfaces 


Services - 


Data 


Operating 

Systems 
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Tomorrow's Architecture 


co 

LL 


0 

cn 

TO 

u. 

O 
•*-* 

co 

TO 

TO 

Q 


"yy Widget Framework 

aft ag K ** I. .f & 


V<£ 


Web Tier 


Application Services 


Alerting 


Ingest 

Real time Enrichment 


Analytics 

Streaming-based 


Analytics 

Map Reduce-based 



Secure Data Access Services 

Casport, Wavelegal, etc 

Relational Store 

Cloudbase 

Enrichment. User. & Auxiliary Data 

Event Data and Analytic Results 

Java VM 

Java VM 

Operating System 

Hardened RH Linux 

Operating System 

Hardened RH Linux 

OpenStack VM 

Utility Cloud 

Data/Anaiytic Cloud 

Hardware Layer 

Hardware Layer 


The Cloud will bring new data-intensive capabilities, support existing 
missions, and align RTRG installations with emerging NSA and 1C standards 
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Data Storage (HDFS) 






















































UNCLASSIFIED//FOR OFFICIAL USE ONLY 



• RTRG Mission Overview 

• RTRG System: Today and Tomorrow 

• Target-Centric and Network-Centric 
Cloud Analytics 

• Future Work 
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Analytics Overview 

(iSK^iP 

Target Development 

Graph & Network 



Graph Triage 
Handset Swap 

Target Enrichment 
& Disambiguation 


Co-Travelers & 
Meetings 

Contact Similarity 

High Priority 
Untasked Number: 


Beddown 

Layercake 

Heatmaps 


Geo-Spatial 

The data-intensive computing capabilities in the system enables a set of 
graph/network analytics and target development analytics 


16 


TOP SECRET//SI//REL TO USA, FVEY 












TOP SECRET//SI//RELTO USA, FVEY 


Meeting Target Development Challenges 


Past 


Analysts manually queried multiple, independent repositories, 
aggregating results in Excel, taking hours or work for search and refinement 



Now RT-RG provides a streamlined, integrated workflow saving analyst effort 


Tips 

i 

I 

Selectors 

pPi 

r 3 

GIST Report 


□ 


Detainee 

Reports 


Target Development 


Event Type | comnns.gsm ~^| 
Event Subtype | coll * 

Selectors = 


n 


| Ksndoliftf 

Submit Query J [ Cancel Query j | Export To File,.,, 


"3 



(T5//5T//REL TO USA, FVEV) 
K FRCj- AFghanstdn lageU 

ProCeiffic; APOCC 


Pttfit fiaporti Network CHsnoetoS 

OBJ DEWEY BEACH ASSOCIATE 

Pech vdey IN', posstrty n ch =rge of a“company" seed etement. led the at! 
A - r Cj (T5//SI//RR, TO USA. F¥EV}(1MFI> 

D (IS//SI/,'f!EL TO USA. FVEV) (IMtl} 

h‘!- JS i 

□ (T5//SI//REI TQU5A*F¥EVH^T5DN> view report # 

■i 


y 


Metadata Search 



Collaboration 


► 


CST 17 FOB FENTY (PAN)tAF Ih 
(Khakssr) 

Thu. 10 May 2012 IS;55:20 Zu 
TOP SECRET//COMINT//REL TO 
NZL//203201QS 

ASSET: JUGGERNAUT 



Geospatial Alerting 


Enrichment 


Outcomes 



Geolocation 



Capture 
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Target Development with Meetings 


Who is at the same UCELLID at the same time? 





Manual Process 

• Take your selector and query for every unique 
location he has been and at what time 

• Query for other selectors who have been at the 
same places at the same times (impossible or 
painful) 

• OR compare to another known set of selectors to 
find overlap (excel / ArcGIS / JEMA) (limiting to 
what you know) 

• Summary statistics on the matching IMSIs using 
excel or ArcGIS 


Cloud Process 

• Pre-calculates all UCELLID overlaps between 
tasked selectors 

• Simply query your selector in cloud-generated 
QFD and view summary statistics 
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Target Development with Co-Travelers* 


Is there a pair traveling together? 




Manual Process 

* You could use the same manual process from 
Meetings, however, this would not find co¬ 
travelers on different networks 

* Manual comparison of pairs of known selectors is 
possible with ArcGIS or similar spatial tools - You 
must know the pairs up front 


Cloud Process 

* Measures miles-per-hour (MPH) between tasked 
selectors as they move around. 

* Low average MPH = co-traveling. 

* Simply query for your selector to view statistics 
on average MPH, days calculated, etc. 


*Also known as "Sidekicks" 
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Meeting Network Challenges 



Post Manually query multiple repositories and build network with Analyst 
Notebook (ANB) - amount of labor can be prohibitive 

Now RT-RG tools exist for contact chaining for selector-to-selector & 
selector-to-report graphs, with more analytics and tools to come 



& & *■* 


o 





Selector-to-Report Graph from Enrichment MAINWAY graph in RTRG Ul 
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Graph Analysis with Furious Chainsaw 
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* Result is Furious Chainsaw 


* Graph representation is 
natural for DNR 


- Prototype on Cloudbase 

- Now supports contact chains 
and trends 

- Will support other graph 
analytics in the future 

* Triage capability for forward 
users to complement 
Enterprise databases 

* Enables chaining and other 
analytics, provides foundation 
for graph algorithms 


Metadata matrix in Cloudbase 
supports fast graph traversal 


Graph View in Renoir - but many 
other analytics are possible 

21 
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Graph Triage: Multiple Views 


Fri ends and Farm I y 


Contact Neighborhood 
Ozone Widget 



- % t 




loe loo Assoc^cm 


lu too Ml 


Top Associates 
Ozone Widget 






r 


0 PANOPTICON I QPra*■«’ I Parers 



Deuui& 

Rc4anonsmps 

Discussions 


g GO • 
u 

I 40 - 

LU 


0+1 of 


Daily Call Trends 
in Panopticon 


Oli J45670 9 1011121314151617181920212223 
Hour of Day (Zulu) 


Weekly Call Trends 
Ozone Widget 



DNR graphs in Furious Chainsaw tables in Cloudbase 
support a wide range of fast queries and analytics 
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Unstructured Data Exploitation 




* Selector extraction, normalization, and 
enrichment 

* Flexible free-text query interface 

* Graph, text, and spreadsheet output 
formats 


Structured Knowledge Space 

• Entity extraction (people, organizations, 
times, geos) 

• Keyword, faceted, and people search 

• Document clustering 

• Arabic name expansion 



• Integrated data ingest using Niagara Files (NiFi) 

• SharkQuery: search by selector, entity, location, and keyword 

• SharkDocs: query, sharing, and collaboration on user uploaded documents 

• Visualization of results in query overview, table, graph, and map 

• Cloudbase and HDFS for scalable text analytics platform 
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Target Development - SharkFinn / SKS 



SKS 


"OBJ SMUHERS" AND 
OBJECTIVESiOBJ TOPPER 
Previous 30 Days 


1. Perform keyword 
search with seed OBJ 
on report library 


OBJECTIVES 
OBJ SMUHERS (19} 

OBJ TOPPER (19) 

OBJ SPRINGFIELD (12) 

OBJ APU (6) 

OBJ GROUNDSMAN WILLY (6) 
OBJ WIGGUM (4) 

OBJ MAYFIELD (3) 

OBJ NEVIS (1) 

OBJ TRAJAN (1) 


2. Finding co-occurring 
objectives 



4. Additional 
Search & Filtering 




5. Correlate found 
selectors with 
Octave/UTT 
and Panopticon 


6. Export selector & 
Report Graph 


3. Selector 
co-occurrence 
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Past 

Now 


Meeting Geolocation Challenges 


Analysts manually correlated locations using map viewers or spreadsheets, 
aggregating data from multiple sources 

Analytics and alerts push target information by subscription 



4U.001 wots imi 


□+□♦++ □ 

UCELLIDs 

412 001 OCO?6 1140 2 +++ Q#*+ 




j Day 



+ First Event 
□ Last Event 


RT-RG pattern of life analytic for target location cues 


Analysts are notified by alerts, 
based on: 

• Geospatial NAI* 

• Tasking status 

• SMS content 

• Selectors, callsigns, frequencies 
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Target Geolocation with Bed Down 


Find the most consistent location of the day's first/last event 

M 




■i 

pT7>v 


First 
Events 



Manual Process - One Selector At a Time 

* Query all events for your selector. 

* Mark first and last events manually. 

•OR 

• Enter in a tool like CheekyMonkey to view gaps in 
activity. 

• Slow process to do one selector at a time 


r 

ri _ J 

1 

[c 

Bed 

)owr 

i] 

L 


y 


Cloud Process- All Tasked Selectors 

• Pre-calculates first and last events in local time 
for ALL selectors. 

• Will calculate estimated Bed Down at query time. 

•Can query multiple selectors in seconds, find 
common overlap. 
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Target Geolocation with LayerCake 




Manual Process 

* Query all events for your selectors. 

* Display events spatially on mapping software 
(impossible to view polygon overlaps). 


• Rasterize and do "raster math" to determine max 
overlap, (very complex and expensive task in most 
GIS tools). 


Cloud Process 

* Pre-calculates unique locations visited for ALL 
selectors. 


Raster heat maps drawn at query time. 
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Spatial Visualization of Data 


Using MapReduce, analytics can count and aggregate over large number 
of daily events to give a multi-resolution spatial visualization of the data 




Analytic Data Flow 


Synoptic View 



Call volume map of RTRG (AF1) collection 
(A day in Dec, 2011) 


Fewer DNR events Fewer events 
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Spatial Visualization of Data 


Cloud enabled analytic allows viewing the spatial data at many levels-of-detail 



"Meso-scale" View 



Heatmap of activity of a set of targets 



Detail View 


Mashup" with heatmap 


mm&m i 


We will arwe on Saturday 


Ant) l the day of judgment 
dealing desiring all believers is. 
WS twill tgnhrtjtC'n +% 3hjrd3yt 


commander religion is: light 
heaven/sky candle land is: to Ihe 
returretlitin-dSv R 


Direction^. To here - From here 


Using Cybertrans translation service 
on SMS messages, integrated with heatmaps 
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Fewer calls More calls 
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• RTRG Mission Overview 

• RTRG System: Today and Tomorrow 
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• Improved DNI capabilities; focus on 
convergence 

• Integrating active SIGINT capabilities 

• Increased CT and expeditionary capabilities 

• Better tools for faster analytic development 

• Incorporation of content analysis 
and HLT capabilities 

• Improved integration between target and 
population analytics 
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RTRG Planned Cloud Instances 
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gmGulf 

NIOC-Bahrain 


gmZilla 

Kabul 


gmSeminole 
NSA Georgia 
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* RTRG has been a successful regional data store and 
exploitation system for COIN, CIED and other missions 

* Moving to NSA Cloud infrastructure 

- More historical data 

- Deeper analysis using parallel programs 

- Allows for more flexible deployments to 1C, DoD service 
installations 

* Continuing to support advanced analytics for current and 
future operations 
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